The United States and China will play an important role in navigating safety and security challenges relating to advanced artificial intelligence. We sought to better understand how experts in each country describe safety and security threats from advanced artificial intelligence, extreme risks from AI, and the potential for international cooperation. Specifically, we compiled publicly-available statements from major technical and policy leaders in both the United States and China. We focused our analysis on advanced forms of artificial intelligence, such as artificial general intelligence (AGI), that may have the most significant impacts on national and global security. Experts in both countries expressed concern about risks from AGI, risks from intelligence explosions, and risks from AI systems that escape human control. Both countries have also launched early efforts designed to promote international cooperation around safety standards and risk management practices. Notably, our findings only reflect information from publicly available sources. Nonetheless, our findings can inform policymakers and researchers about the state of AI discourse in the US and China. We hope such work can contribute to policy discussions around advanced AI, its global security threats, and potential international dialogues or agreements to mitigate such threats.

Understanding the potential of generative AI (GenAI)-based attacks on the power grid is a fundamental challenge that must be addressed in order to protect the power grid by realizing and validating risk in new attack vectors. In this paper, a novel zero trust framework for a power grid supply chain (PGSC) is proposed. This framework facilitates early detection of potential GenAI-driven attack vectors (e.g., replay and protocol-type attacks), assessment of tail risk-based stability measures, and mitigation of such threats. First, a new zero trust system model of PGSC is designed and formulated as a zero-trust problem that seeks to guarantee for a stable PGSC by realizing and defending against GenAI-driven cyber attacks. Second, in which a domain-specific generative adversarial networks (GAN)-based attack generation mechanism is developed to create a new vulnerability cyberspace for further understanding that threat. Third, tail-based risk realization metrics are developed and implemented for quantifying the extreme risk of a potential attack while leveraging a trust measurement approach for continuous validation. Fourth, an ensemble learning-based bootstrap aggregation scheme is devised to detect the attacks that are generating synthetic identities with convincing user and distributed energy resources device profiles. Experimental results show the efficacy of the proposed zero trust framework that achieves an accuracy of 95.7% on attack vector generation, a risk measure of 9.61% for a 95% stable PGSC, and a 99% confidence in defense against GenAI-driven attack.

On Tuesday, 24 AI experts, including Turing Award winners Geoffrey Hinton and Yoshua Bengio, released a paper calling on governments to take action to manage risks from AI. The policy document had a particular focus on extreme risks posed by the most advanced systems, such as enabling large-scale criminal or terrorist activities. The paper makes a number of concrete policy recommendations, such as ensuring that major tech companies and public funders devote at least one-third of their AI R&D budget to projects that promote safe and ethical use of AI. The authors also call for the creation of national and international standards. Bengio, scientific director at the Montreal Institute for Learning Algorithms, says that the paper aims to help policymakers, the media, and the general public "understand the risks, and some of the things we have to do to make [AI] systems do what we want."

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.

At several points in human history, there have been great transitions that accelerated our progress and shaped everything that would follow. Ten thousand years ago, the first Agricultural Revolution allowed us to establish cities and civilisation. Four hundred years ago, the Scientific Revolution gave us a reliable method for understanding the laws of nature. Two hundred years ago, the Industrial Revolution launched us on a new trajectory of rapid economic growth. But there has recently been another transition more important than any that has come before.

Disease warfare, leaks of dangerous pathogens from labs and misuse of artificial intelligence could be much more dangerous than the COVID pandemic, a report has warned. The UK government should now be making "serious efforts" to guard against such "extreme risks", say the authors of the Future Proof study. They warn such threats have the potential for huge loss of life globally - and at their worst could even "lead to the premature extinction of humanity". There is a one in six "Russian roulette" chance of an "existential catastrophe over the next one hundred years", according to one of the report's lead authors. Oxford and Cambridge University researchers worked on the study for The Centre for Long-Term Resilience and say it is a "once-in-a-generation opportunity" for the UK to beef up its protections.

It is profoundly difficult to grapple with risks whose stakes may include the global collapse of civilisation, or even the extinction of humanity. The pandemic has shattered our illusions of safety and reminded us that despite all the progress made in science and technology, we remain vulnerable to catastrophes that can overturn our entire way of life. These are live possibilities, not mere hypotheses, and our governments will have to confront them. As Britain emerges from Covid-19, it could find itself at the forefront of the response to future disasters. The government's recent integrated review, Britain's taking of the G7 presidency and the Cop26 climate conference, which will be hosted in Glasgow later this year, are all occasions to address global crises. But in order to ensure that the UK really is prepared, we need to first identify the biggest risks that we face in the coming decades.